Automated template layout method

ABSTRACT

A method for producing an image template having one or more openings for displaying images. The openings as well as the images are each associated with a season. The images are automatically analyzed to determine which season they most likely depict. Images are then selected to be placed in a template opening designated for a particular season. This method can be used to automate calendar production.

CROSS REFERENCES TO RELATED APPLICATIONS

U.S. patent application Ser. No. 12/______, entitled “AUTOMATED TEMPLATELAYOUT SYSTEM”, filed concurrently herewith is assigned to the sameassignee hereof, Eastman Kodak Company of Rochester, N.Y., and containssubject matter related, in certain respect, to the subject matter of thepresent application. The above-identified patent application isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to computer-implemented personalization ofimage products employing semantic analysis of digital images.

BACKGROUND OF THE INVENTION

Calendars are a widely used device for tracking the passage of time, fororganizing activities, and providing reminders. Many calendars areprinted on bi-fold paper with a grid of dates marked in weeks on onesheet and a decorative image printed on the other. The grid of dates isfrequently large enough to accommodate written schedule reminders.

Most calendars include a year of dates and are organized around monthsand present a month of dates together with a decorative element, usuallyan image or graphic. Other calendars record a time span other than ayear, for example a month or a season, for example the competitiveseason for a sporting team, or for a holiday season, such as theChristmas and New Year's holidays celebrated in many countries.

Calendars for a new year are often given as year-end gifts from oneperson to another and are personalized with images reminding the giftreceiver of the gift giver. For example, children can make a calendarthat includes pictures of their family and their family's activitiesthroughout the year and give it to their grandparents.

In the past, creating such personalized calendars required significantwork. Image selection, copying, and drawing calendars were expensive andlabor-intensive. In recent years, calendar kits have become availablethat ease the task. In some cases, the kits provide preprinted calendarswith suitable mounts for individual images. In other cases, computersoftware and layout tools are used together with digital images tocreate a personalized calendar. Nonetheless, the calendar creation taskrequires significant effort, particularly for the selection and layoutof suitable images. This issue is becoming increasingly problematic, asthe number of digital images that consumers make is becoming very largeand reviewing them is increasingly tedious.

U.S. Pat. No. 5,986,671 describes a method of combining digital imageswithin a template and illustrates the use of the method with a calendar.U.S. Pat. No. 6,186,553 and U.S. Pat. No. 7,090,253 describe theconstruction of a theme calendar using computer-aided layout tools andcomputer stored artwork. However, none of these prior-art methodsaddress one of the central difficulties of personalized calendarcreation, the selection of suitable images from a potentially large setof images.

In U.K. Patent Application GB2403304A, Rowe describes a method oflabeling images with labels based on the image capture datescorresponding to national events (and seasons) for later use intext-based search and retrieval of images. However, not all images havereliable metadata for dates nor are all suitable for, or representativeof, a desired season.

U.S. Patent Application 2007/0177805 describes a method of searchingthrough a collection of images, includes providing a list of individualsof interest and features associated with such individuals; detectingpeople in the image collection; determining the likelihoods for eachlisted individual of appearing in each collection image in response tothe detected people and the features associated with the listedindividuals; and selecting in response to the determined likelihoods anumber of collection images such that each individual from the listappears in the selected number of collection images. This enables a userto locate images of particular people but does not necessarily assist infinding suitable images for a particular season.

U.S. Pat. No. 7,271,809 describes a method for using viewing time todetermine affective information in an imaging system that is employed toestimate user preferences for an image. This enables a user to locatepreferred images but does not necessarily assist in finding suitableimages for a particular season.

There is a need, therefore, for an improved method for personalizing animage product and making suitable image selections for the product.

SUMMARY OF THE INVENTION

In accordance with a preferred embodiment of the present invention,there is provided a computer implemented method for generating andstoring a list of elements. The elements are objects found in, orcharacteristics of, a digital image. Each of the elements includes anassociated season and a weight value indicating a strength ofassociation between each element and its associated season. The methodcan be advantageously used by searching through digital images and, ifan element is found in a digital image, a season identifier can beassociated with the digital image. The season identifier can be storedwith the digital image and later used to identify images associated witha particular season.

A digital image analysis step includes detecting a presence of one ormore of the elements in each of the digital images, calculating aconfidence value for each element detected in digital images, andcalculating a season determination value for each of the digital images.The season determination value is based on the weight value for each ofthe elements found in the digital images and on a confidence value foreach element detected. In this manner, an image is associated with aseason and that determination is stored with the digital image for lateruse. Of course, each image will have many detected elements and eachelement will have a confidence value and a weight value associated witha particular season. These weights and values are used to calculate afinal season determination value. This is the value that is stored withthe digital image and is used to associate the image with a season.

In accordance with another preferred embodiment of the presentinvention, there is provided a computer implemented method for producingan image product by providing a template with openings for displayingdigital images. Each opening is used for displaying an image depicting aparticular season. By analyzing the pixels of the digital image theseason depicted by the digital image is determined. The images displayedare first sorted into one or more seasonal groups. A digital image isselected from a seasonal group having the same season as is assigned toa corresponding template opening and a template is produced with thedigital images.

In accordance with another preferred embodiment of the presentinvention, there is provided a computer implemented method for producingan image product. By analyzing the pixels of the digital images, aseason depicted by the digital images is determined. A template havingone or more openings corresponding to the determined season is selected.For each template opening, a selected digital image depicting a seasonintended for a particular template opening is displayed therein.

Preferred embodiments of the present invention have the advantage thatthe process of personalizing an image product is made simpler, faster,and provides a more satisfactory result. These, and other, aspects andobjects of the present invention will be better appreciated andunderstood when considered in conjunction with the following descriptionand the accompanying drawings. It should be understood, however, thatthe following description, while indicating preferred embodiments of thepresent invention and numerous specific details thereof, is given by wayof illustration and not of limitation. For example, the summarydescriptions above are not meant to describe individual separateembodiments whose elements are not interchangeable. In fact, many of theelements described as related to a particular embodiment can be usedtogether with, and possibly interchanged with, elements of otherdescribed embodiments. Many changes and modifications may be made withinthe scope of the present invention without departing from the spiritthereof, and the invention includes all such modifications. The figuresbelow are intended to be drawn neither to any precise scale with respectto relative size, angular relationship, or relative position nor to anycombinational relationship with respect to interchangeability,substitution, or representation of an actual implementation.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentinvention will become more apparent when taken in conjunction with thefollowing description and drawings wherein identical reference numeralshave been used, where possible, to designate identical features that arecommon to the figures, and wherein:

FIG. 1 illustrates a computer system for use in a preferred embodimentof the present invention.

FIG. 2 illustrates a user implementing a computer system in a preferredembodiment of the present invention.

FIG. 3 is a flow chart of a method according to an embodiment of thepresent invention;

FIG. 4 is a schematic illustration of image products made according to amethod of a preferred embodiment the present invention;

FIGS. 5A and 5B are schematic illustrations of alternative imageproducts made according to a method of a preferred embodiment of thepresent invention;

FIG. 6 is a flow graph of a method according to a preferred embodimentof the present invention;

FIG. 7 is a flow graph of a portion of a method according to a preferredembodiment of the present invention;

FIG. 8 is a flow graph of a method according to an alternativeembodiment of the present invention;

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a first embodiment of an electronic system 26, acomputer system, for implementing certain embodiments of the presentinvention for automatically generating image enhanced products. In theembodiment of FIG. 1, electronic computer system 26 comprises a sourceof content and program data files 24 such as software applications,calendar template files, calendar designs, association sets, imagefiles, and image season information, which includes various memory andstorage devices 40, a wired user input system 68 as well as a wirelessinput system 58, and an output system 28, all communicating directly orindirectly with processor 34. Although not shown processor 34 is meantto illustrate typical processor system and chip components such asinstruction and execution registers, an ALU, various levels of cachememory, etc. The source of program and content data files 24, user inputsystem 68, or output system 28, and processor 34 can be located within ahousing (not shown). In other embodiments, circuits and systems of thesource of content and program data files 24, user input system 68 oroutput system 28 can be located in whole or in part outside of ahousing.

The source of content or program data files 24 can include any form ofelectronic, optical, or magnetic storage such as optical discs, storagediscs, diskettes, flash drives, etc., or other circuit or system thatcan supply digital data to processor 34 from which processor 34 can loadsoftware, calendar template files, calendar designs, association sets,image files, and image season information, and derived and recordedmetadata. In this regard, the content and program data files cancomprise, for example and without limitation, software applications, astill image data base, image sequences, a video data base, graphics, andcomputer generated images, image information associated with still,video, or graphic images, and any other data necessary for practicingembodiments of the present invention as described herein. Source ofcontent data files 24 can optionally include devices to capture imagesto create image data files by use of capture devices located atelectronic computer system 20 and/or can obtain content data files thathave been prepared by or using other devices, or image enhancement andediting software. In the embodiment of FIG. 1, sources of content orprogram data files 24 includes sensors 38, a memory and storage system40 and a communication system 54.

Sensors 38 can include one or more cameras, video sensors, scanners,microphones, PDAs, palm tops, laptops that are adapted to capture imagesand can be coupled to processor 34 directly by cable or by removingportable memory 39 from these devices and/or computer systems andcoupling the portable memory to slot 46. Sensors 38 can also includebiometric or other sensors for measuring physical and mental reactions.Such sensors including, but not limited to, voice inflection, bodymovement, eye movement, pupil dilation, body temperature, and p4000 wavesensors.

Memory and storage 40 can include conventional digital memory devicesincluding solid state, magnetic, optical or other data storage devices,as mentioned above. Memory 40 can be fixed within system 26 or it can beremovable and portable. In the embodiment of FIG. 1, system 26 is shownhaving a hard disk drive 42, which can be an attachable external harddrive, which can include an operating system for electronic computersystem 26, and other software programs and applications such as theprogram algorithm embodiments of the present invention, a templatedesign data base, derived and recorded metadata, image files, imageattributes, software applications, and a digital image data base. A diskdrive 44 for a removable disk such as an optical, magnetic or other diskmemory (not shown) can also include control programs and softwareprograms useful for certain embodiments of the present invention, and amemory card slot 46 that holds a removable portable memory 48 such as aremovable memory card or flash memory drive or other connectable memoryand has a removable memory interface 50 for communicating with removablememory 48. Data including, but not limited to, control programs,template designs, derived and recorded metadata, digital image files,image attributes, software applications, digital images, and metadatacan also be stored in a remote memory system 52 such as a personalcomputer, computer network, a network connected server, or other digitalsystem.

In the embodiment shown in FIG. 1, system 26 has a communication system54 that in this embodiment can be used to communicate with an optionalremote input 58, remote memory system 52, an optional remote display 56,for example by transmitting image designs in the form of calendartemplate designs with or without merged images and receiving from remotememory system 52, a variety of control programs, template designs,derived and recorded metadata, image files, image attributes, andsoftware applications. Although communication system 54 is shown as awireless communication system, it can also include a modem for couplingto a network over a communication cable for providing to the computersystem 26 access to the network and remote memory system 52. A remoteinput station including a remote display 56 and/or remote input controls58 (also referred to herein as “remote input 58”) can communicate withcommunication system 54 wirelessly as illustrated or, again, cancommunicate in a wired fashion. In a preferred embodiment, a local inputstation including either or both of a local display 66 and local userinput controls 68 (also referred to herein as “local user input 68”) isconnected to processor 34 which is connected to communication system 54using a wired or wireless connection.

Communication system 54 can comprise for example, one or more optical,radio frequency or other transducer circuits or other systems thatconvert data into a form that can be conveyed to a remote device such asremote memory system 52 or remote display 56 using an optical signal,radio frequency signal or other form of signal. Communication system 54can also be used to receive a digital image and other data, asexemplified above, from a host or server computer or network (notshown), a remote memory system 52 or a remote input 58. Communicationsystem 54 provides processor 34 with information and instructions fromsignals received thereby. Typically, communication system 54 will beadapted to communicate with the remote memory system 52 by way of acommunication network such as a conventional telecommunication or datatransfer network such as the internet, and peer-to-peer; cellular orother form of mobile telecommunication network, a local communicationnetwork such as wired or wireless local area network or any otherconventional wired or wireless data transfer system.

User input system 68 provides a way for a user of system 26 to provideinstructions to processor 34, such instructions comprising automatedsoftware algorithms of particular embodiments of the present inventionthat automatically generate coordinated image templates according toselected template designs. This software also allows a user to make adesignation of content data files, such as selecting calendar templatesand designating digital image files, to be used in automaticallygenerating an image enhanced output calendar product according to anembodiment of the present invention and to select an output form for theoutput product. User controls 68 a, 68 b or 58 a, 58 b in user inputsystem 68, 58, respectively, can also be used for a variety of otherpurposes including, but not limited to, allowing a user to arrange,organize and edit content data files, such as coordinated image displaysand calendar image templates, to be incorporated into the image outputproduct, for example, by incorporating image editing software incomputer system 26 which can be used to override design automated imageoutput products generated by computer system 26, as described below incertain preferred method embodiments of the present invention, toprovide information about the user, to provide annotation data such astext data, to identify characters in the content data files, and toperform such other interactions with system 26 as will be describedlater.

In this regard user input system 68 can comprise any form of devicecapable of receiving an input from a user and converting this input intoa form that can be used by processor 34. For example, user input system68 can comprise a touch screen input 66, a touch pad input, a multi-wayswitch, a stylus system, a trackball system, a joystick system, a voicerecognition system, a gesture recognition system, a keyboard 68 a, mouse68 b, a remote control or other such systems. In the embodiment shown inFIG. 1, electronic computer system 26 includes an optional remote input58 including a remote keyboard 58 a, a remote mouse 58 b, and a remotecontrol 58 c. Remote input 58 can take a variety of forms, including,but not limited to, the remote keyboard 58 a, remote mouse 58 b orremote control handheld device 58 c illustrated in FIG. 1. Similarly,local input 68 can take a variety of forms. In the embodiment of FIG. 1,local display 66 and local user input 68 are shown directly connected toprocessor 34.

As is illustrated in FIG. 2, computer system 26 and local user inputsystem 68 can take the form of an editing studio or kiosk 70 (hereafteralso referred to as an “editing area 70”), although this illustration isnot intended to limit the possibilities as described in FIG. 1 ofediting studio implementations. In this illustration, a user 72 isseated before a console comprising local keyboard 68 a and mouse 68 band a local display 66 which is capable, for example, of displayingmultimedia content. As is also illustrated in FIG. 2, editing area 70can also have sensors 38 including, but not limited to, camera or videosensors 38 with built in lenses 89, audio sensors 74 and other sensorssuch as, for example, multispectral sensors that can monitor user 72during a user or production session.

Output system 28 (FIG. 1) is used for rendering images, text, completedor uncompleted digital image templates and other digital image outputproducts, or other graphical representations in a manner that allows animage output product to be generated. In this regard, output system 28can comprise any conventional structure or system that is known forprinting, displaying, or recording images, including, but not limitedto, printer 29. For example, in other embodiments, output system 28 caninclude a plurality of printers 29, 30, 32, and types of printers,including transfer machines capable of screen printing t-shirts andother articles. Processor 34 is capable of sending print commands andprint date to a plurality of printers or to a network of printers. Eachprinter of the plurality of printers can be of the same or a differenttype of printer, and each printer may be able to produce prints of thesame or a different format from others of the plurality of printers.Printer 29 can record images on a tangible surface, such as on, forexample, various standard media or on clothing such as a T-shirt, usinga variety of known technologies including, but not limited to,conventional four color offset separation printing or other contactprinting, silk screening, dry electrophotography such as is used in theNexPress 2100 printer sold by Eastman Kodak Company, Rochester, N.Y.,USA, thermal printing technology such as in thermal printer 30, drop ondemand ink jet technology and continuous inkjet technology. For thepurpose of the following discussions, printers 29, 30, 32 will bedescribed as being of a type that generates color images. However, itwill be appreciated that this is not necessary and that the claimedmethods and apparatuses herein can be practiced with printers 29, 30, 32that print monotone images such as black and white, grayscale or sepiatoned images.

In certain embodiments, the source of content data files 24, user inputsystem 68 and output system 28 can share components. Processor 34operates system 26 based upon signals from user input system 58, 68,sensors 38, memory 40 and communication system 54. Processor 34 caninclude, but is not limited to, a programmable digital computer, aprogrammable microprocessor, a programmable logic processor, a series ofelectronic circuits, a series of electronic circuits reduced to the formof an integrated circuit chip, or a series of discrete chip components.

Referring to FIG. 3, a flow chart describing a computer implementedmethod of assigning a digital image to a template opening isillustrated. In step 300, an association set, such as described belowwith reference to Table 1 and Table 2, is accessed by the computersystem. The association set can be previously stored in the computersystem or provided by a user via portable memory or otherwise accessibleover a local or wide area network or over the internet by computersystem 26. A digital template with one or more openings each with anassociated season is accessed and selected in step 301. Similar to thestep of accessing an association set, the digital template can beselected from a group of previously stored digital templates in thecomputer system or provided by a user via portable memory or otherwiseaccessible over a local or wide area network or over the internet bycomputer system 26. Each opening in the digital template is associatedwith a season based on data stored with the template file which alsocontains a digital image of the template for displaying on systemdisplay 56, 66. An image set comprising digital images from whichsuitable images are to be found is selected in step 305. Similar to thestep of accessing an association set and a digital template, the digitalimages are selected from a group of previously stored digital images inthe computer system or provided by a user via portable memory orotherwise accessible over a local or wide area network or over theinternet by computer system 26. In step 310, each image is analyzed todetermine the best season match for that image. In order to calculatesuch a match, well known algorithms for identifying objects, colors,textures, or shapes appearing in each image are utilized in step 306.Although not described in detail herein, such algorithms are describedin, for example, Digital Image Processing: PIKS Scientific Inside byWilliam K. Pratt, 4th edition, copyright 2007 by John Wiley and Sons,ISBN: 978-0-471-76777-0, and U.S. Pat. No. 6,711,293, to Lowe, whichdefines an algorithm for object recognition and an aggregate correlationthat is useable as a confidence value, which is incorporated herein byreference in its entirety. The result of the algorithms includes aconfidence value that a detected object, color, texture, or shape ineach digital image is accurately identified. Table 1, in which eachElement in the association set is searched for in each digital image,provides a list of Elements to search for (first column) as well astable cells for entering the results of the search. Thus, a preferredembodiment of the present invention includes the step of reading thetable entries under the Elements column and, for each Element, appliesthe well known object identification algorithms identified above tocalculate for each Element a confidence value (C_(i)) that an object,color, texture, or shape corresponding to the current Element has beendetected in the current digital image. The value is entered in the tablefor that particular Element.

The table separately charts a prevalence value (P_(i) or P_(ij)) foreach season corresponding to each Element which indicates a strength ofassociation between the Element and the season. This prevalence value isseparately determined and can be provided in the table and stored in thecomputer system. The prevalence values can be determined in a variety ofways. They can be calculated based on historical searches of largenumbers of digital images, or they can be entered and stored byindividuals providing a subjective value that indicates an associationbetween such an Element in an image and its correspondence to a season.For example, a detected beach scene can have a high prevalence value forthe season “Summer” or for the holiday season “4th of July” and a lowprevalence value for the season “Winter” or for the holiday season“Christmas.” Such prevalence values are compiled and stored with thetable. Some Elements may have an association of zero with a particularseason. Other Elements may have a varying value for every season columnlisted. An Elements having an equal prevalence value for each seasonlisted in the columns would not serve to differentiate the current imagefor association with a season. Stored prevalence values can be reused asdesired by a user. The user can also enter such prevalence values to bestored in the association set. In this case, a user who is familiar withhis or her collection of digital images can enter realistic prevalencevalues for each season for Elements appearing in his or her imagecollection which will result in more accurate season identifications forhis or her image collection.

Continuing with the algorithm for implementing step 310, the Table 1cells can now be calculated and final values entered therein using Eqn.1 as shown below. In a preferred embodiment of the present invention,the confidence value for each Element is multiplied by the prevalencevalue for each season to determine the Table 1 cell values Wseason. Thepreferred embodiment of the present invention is not limited only tothis algorithm. Table 1 can be easily constructed as a multi-dimensionaldata structure to include more inputs for calculating cell values. Thus,the formula for determining Wseason can be implemented using Eqn. 3shown below. As an example, a user's image collection that includesmetadata that identifies user favorite images can be used as input tothis equation and a resulting Wseason value will be increased for userfavorite images. Other image values can also be included for suchcalculations. These inputs can be optionally used for Table 1 or forTable 2, as described below. After all Elements have been searched forin the digital image set, or in a user selected group of digital images,under consideration, the Total Wseason values are added for each columncorresponding to a season as shown in the last row of Table 1.

The Total Wseason values entered into Table 1 are used in step 320 forpopulating Table 2. Each row in Table 2 corresponds to each image underconsideration and contains the Total Wseason value obtained for aparticular image from step 310. The last column of Table 2 is used toidentify which season, of the seasons identified in the first row, isbest associated with a particular image listed in the first column. Thelast row of Table 2 is used to identify which image, of the imagesidentified in the first column, is best associated with a particularseason listed in the first row. These are simply the highest valueobtained from the respective row and column. Images tagged as userfavorites can optionally be weighted more heavily and the inputs forthose tags used when calculating the Max values in Table 2, rather thanusing them in calculating Table 1 cell values.

In step 325, the image with the largest value from Table 2 is selectedas best representing the season associated with the template openingunder consideration and is thereby associated with the template opening.This association can be implemented as a field entry in the templatefile containing the digital template image. Such association can berecorded in the digital template file as stored in the computer systemor other storage location. An optional step, step 326, includes the stepof ranking multiple images for each season according to its calculatedvalues as provided in Table 2. Preference for inclusion in a template isthen given to the higher valued images in step 325. The resultingweighting can be used, as described above, to order the digital imagesin a seasonal group (e.g. the columns in Table 2), so that the digitalimage with the highest weighting is preferred. In order to implement thepresent method for constructing a calendar, multiple templates areaccessed, for example, one for each month, and the next template to becompleted is accessed in step 301, and the process repeats until amost-representative image has been found in the image set for the seasonassociated with each template opening and until the calendar iscompleted. Completed portions are stored in a calendar file until theentire calendar is completed.

Referring to FIGS. 4, 6, and 8, a method for producing a digital imageproduct comprises the step 600 of a computer system providing a digitaltemplate 410 for display on a computer system 26 display 66 fordigitally placing into one or more digital template openings 412 one ormore digital images, each opening 412, 412A, 412B, 412C corresponding toa season. Data for associating each digital opening with a season, forexample, Spring, Summer, Fall, Winter, is stored in a template filewhich also contains, or is associated with, the template image. One ormore digital images having pixels are provided by a user of the computersystem in step 605, 805. The computer system reads image filescontaining the digital images. These can be coupled to the computersystem by the user via various storage devices capable of storing imagefiles. At a high level, the present computer implemented method includesanalyzing the pixels of the digital images in step 610, 810 to determinea season depicted by the digital images. In step 625, the digital imagesare sorted into one or more seasonal groups corresponding to thedetermined seasons. For each template opening, a digital image isselected from a seasonal group having the same season as the templateopening in step 620, 820. In step 645, 845, the template with thedigital images is produced as an image product. The details of anembodiment of this computer-implemented method will now be described.

The image product includes customized or un-customized text 414 and hasone or several image openings, 412, 412A, 412B, 412C on one or morepages of the template. Thus, the template has one or a plurality ofpages to which the method of the present invention can be applied. Imageopenings in the templates can overlap, form collages, or are in aportrait or landscape configuration. A grid of dates 420, for calendarimage templates such as those of FIG. 4, is included on the bottom halfof a bi-fold page folded across a center line 424. Such arrangements areknown in the design art and such various designs and arrangements arecontemplated for preferred embodiments of the present invention.

Therefore, the step 600 includes the computer-implemented step ofproviding a template having a plurality of openings corresponding to aplurality of seasons. Alternatively, the computer-implemented step 600of providing a template includes the step of providing a template with asingle opening having a single season associated therewith and the stepof providing a template with a plurality of openings, each openinghaving a single season associated therewith.

Another preferred embodiment of the present invention includes theoptional step 615, 815 of comparing the determined season stored inassociation with each of the digital images, via the method describedbelow, to date or location data associated with the digital images thatare also included as metadata stored in association with each digitalimage file. Digital cameras often include software that providesmetadata associated with captured images that record details concerningthe image capture, such as camera settings, the date of capture, and thelocation of capture, either through automated devices (e.g. an internalclock or global positioning system) or via user input. In anotherpreferred embodiment of the present invention, metadata associated witheach image is included in the step 620, 820 of determining the season ofa digital image, wherein the metadata is read by the computer system anda corresponding season is associated with the digital image based onsuch metadata.

An associated date can be associated with a season. This associationcould be a simple month-to-season correspondence. Location informationcan also be used to improve accuracy when determining a season based ondate information. Note however, that for some image products, the datemay not be an adequate predictor of the suitability of a digital imagefor an image product. For example, it is desired to provide an imagethat is representative of a season. However, an image taken at a timeduring the season is not necessarily representative of the season. It isalso possible that the date may be incorrect. Thus the associatedmetadata date is helpful in selecting a suitable image but is notnecessarily indicative or completely definitive.

Similarly, an associated location can be associated with a season,especially in combination with a date. For example, it may be known thata location is associated with a season (e.g. a person is often in aparticular place during a particular season). Hence, images associatedwith the place are associated with the season. As with the date,however, such association does not necessarily mean that an image issuitable to represent a season for a particular image product,particularly if it is desired that the image be representative of aseason.

Once the season of an image is determined, it is sorted (step 625) intoone or more seasonal groups corresponding to the determined seasons. Inthe simplest case, a single seasonal group has only one member, a singleimage. For example, it may be desired simply to determine whether adigital image corresponds to a desired season that is itself associatedwith a product template opening. In this case, the sorting is by defaultbecause there is only one candidate image and requires no action or listconstruction. Such a case is considered to satisfy a sorting step and isincluded in a preferred embodiment of the present invention. In morecomplex situations, for example in creating a one-year calendar, aplurality of images are examined and determined to belong to a pluralityof seasonal groups, each group of which could include multiple images.In another preferred embodiment of the present invention, the images ina seasonal group are ranked (step 630) by image quality, userpreferences, or the degree to which the image is representative of aseason, or some desired combination of these characteristics. This isdescribed in more detail below with reference to the valuationcalculations. A variety of metrics can be employed to order, rank, orsort the images in order of image quality, for example, sharpness andexposure. Affective metrics (such as a user's favorite images, asdetermined by other well-known means or, known by a user's identifyingand storing particular images as favorites) are employed in making theimage selection (step 635) as well. Thus, desired digital images thathave a greater quality than digital images having a lesser quality arepreferentially selected.

The method of a preferred embodiment of the present invention includesthe step of producing the template with the selected digital imagesdisposed in the openings thereof. This is implemented by digitallycompositing the digital images into the digital openings of the pages ofthe digital template (step 640, 840). Alternatively, printed images arephysically mounted into or onto a product. The template with the digitalimages can be printed, stored, displayed, sold, or transported. As usedherein, a template is also considered a product and includes the producttype as well as image or design-related features of the product.

Images representing a variety of seasons can be employed with apreferred embodiment of the present invention. Typical seasons includeweather-related seasons of the year, for example winter, spring, summer,autumn (fall), dry season, rainy (wet) season, monsoon season, and soforth. Holiday seasons can also be represented, for example Christmas,Hannukah, New Year's Valentine's Day, National Day (e.g. July 4 in theUnited States), and Thanksgiving. Seasons include personal holidays orcelebrations, including birthdays and anniversaries.

The analysis step (610, 810) of a method of a preferred embodiment ofthe present invention is facilitated by providing an association set,such as depicted in Table 1, that includes Elements such as objects,colors, textures, or shapes that might be found in a digital imageundergoing analysis for selective placement in a template. Each object,color, texture, or shape listed in the Element column of Table 1 has anassociated prevalence value corresponding to each of a number ofseasons, also listed individually in columns corresponding to eachseason. Thus, an object listed in the first column of elements has aplurality of prevalence values listed in the row to the right of theElement indicating its magnitude of correlation to each particularseason. For example, if an association set includes “Christmas tree” inits column of Elements a corresponding prevalence value under a “Winter”season column will be higher than its prevalence value under a “Summer”season column. Similarly, if a plurality of Season columns includesholiday seasons, then an image having a detected Christmas tree willhave a higher prevalence value in its Christmas season column than inits Easter season column. This association set is formed by ethnographicor cultural research, for example by displaying a large number of imagesto members of a cultural group. The members then relate objects found ineach scene to each season and ranking the object importance to provideprevalence values. The aggregated responses from many respondents canthen be used to populate the association set.

During an analysis step, the programmed computer system accesses apreviously stored association set and searches each digital image forElements identified therein. If an object, color, texture, or shape isfound within a digital image that is in the association set, the digitalimage is scored with respect to each of the seasons that mightcorrespond with the found Element. The resulting score is the prevalencevalue as between the found object (Element) and the Season (column)under analysis. Various Elements listed in the association set may befound in each of a plurality of images, resulting in Total Prevalencevalues that are the sum of prevalence values in each Season column. TheSeason column having the highest Total Prevalence value is the Seasonassociated with a particular image. Such scored images are sorted andstored into seasonal groups by assigning the digital images to theseasonal group corresponding to its associated season.

The following list provides some association sets useful forimplementing the analysis step in different countries or cultures. Notethat different cultures have widely differing associations, so that anassociation set is culturally dependent. The color white can beassociated with winter, Christmas, anniversaries, weddings, and death.The color green can be associated with Christmas, Spring, St. Patrick'sDay, and Summer. The color red can be associated with Christmas,Valentine's Day, and National Day. The color orange can be associatedwith autumn, thanksgiving, and National Day. Combinations of colors areassociated with a season, for example red, white, and blue are thenational colors of several countries and are associated with thosecountries' National Day. Flesh tones can be associated with summer, andseasons can be associated with digital images containing people, forexample anniversaries and birthdays in which images of people areprevalent. Objects and displays can be part of association sets:Fireworks can be associated with summer, National Day, and New Year'sDay, while candles can be associated with birthdays, anniversaries, andpersonal celebrations. Snow can be associated with winter and Christmasin northern climates, while green grass can be associated with springand summer. Water can be associated with summer and holidays whileflowers can be associated with anniversaries and Spring. According to apreferred embodiment of the present invention, association sets are notlimited to the foregoing examples.

As these examples make clear, associating a digital image with a seasoninvolves a number of calculations as well as evaluating the metadatadiscussed above. A plurality of objects, colors, textures, or shapeslisted in the association set can be found in a single digital image.Furthermore, an object, color, texture, or shape can be associated withmore than one season.

Nonetheless, prevalence value results define which season or seasons aremost highly associated with a particular image. In the event that animage is equally associated with a plurality of different seasons in anassociation set, a random method can be used to categorize the imageinto one of the seasons. Another option is to weight particular Elementsas more indicative of a season and select a highest prevalence value ofone of the Elements as the associated season.

The confidence value is an accuracy indicator of how likely the foundelement really is the element recognized and the prevalence valueindicates how strongly the element is associated with the season.

The size of the element and the location of the element within the imagealso affect the prevalence value so that, in a preferred embodiment ofthe present invention, the prevalence value is a function rather than asingle number. If both the confidence and prevalence values are low, theweight given to the seasonal assignment is likewise low. If both theconfidence and prevalence values are high, the weight given to theseasonal assignment is high. In a preferred embodiment of the presentinvention, the weight value is a product of the confidence value and theprevalence value, as described in more detail below.

For example, a seasonal assignment weight value for a digital image fora given season is expressed as:

Wseason=ΣCi*Pi  Eqn. 1

where Ci is the confidence value that each found element i in thedigital image is the matched element in the association set and Pi isthe prevalence value for each found element in the association set foreach season. A C value can be determined using image processingcalculations known in the image processing art. For example, a veryspecific object of a known size can be found by a two-dimensionalconvolution of an object prototype with a scene. The location of thelargest value of the convolution represents the location of the objectand the magnitude of the value represents the confidence that the objectis found there. More robust methods include scale-invariant featuretransforms that use a large collection of feature vectors. Thisalgorithm is used in computer vision to detect and describe localfeatures in images (see e.g. U.S. Pat. No. 6,711,293 entitled “Methodand apparatus for identifying scale-invariant features in an image anduse of same for locating an object in an image”, identified above). Analternative method can employ Haar-like features. Thus, Elements thatare not found in the digital image have a C value of zero. Elements thatare found in the digital image with a high degree of certainty, orconfidence, have a C value of nearly 1. If the found element is highlycorrelated with a season, the P value is high. If the found element isnot correlated with a season, the P value is low. The calculation isrepeated for each Element for each season under evaluation. Each digitalimage under evaluation is analyzed and sorted into the seasonal groupcorresponding to the highest Wseason value. The images within eachseasonal group are then ranked within the seasonal group by theirWseason values. The digital image with the highest Wseason value withina seasonal group is the preferred digital image for that season, e.g.

Pref_(group)=MAX(Wseason)  Eqn. 2

The preferred image within a group is thus the image with the highestWseason ranking and is selected for a corresponding-season templateopening. As mentioned previously, if two images have equal Wseasonvalues, a random selection procedure or a weighted selection procedure(e.g. preferred Element value) can be implemented to select a digitalimage.

The ranking can also include additional parameters or factors such asdate and location correlation, or user preference (favorites). Forexample,

Wseason=ΣCi*Pi*Di*Li*Fi  Eqn. 3

where Di is a date matching metric, Li is a location matching metric,and Fi is a preference matching metric. The Di value can be obtainedfrom image capture devices that include clocks such as some digitalcameras or by user input. The Li value can be obtained from imagecapture devices that include global positioning systems such as somedigital cameras or by user input. The Fi value can be obtained from userinput or records of image use, the more frequently used images beingpresumed to be favored.

While the combinations shown in the equations above are multiplicative,other combination formulas are possible, for example linear or acombination of linear and multiplicative.

In a preferred embodiment of the present invention, the association setis organized as a table, and a table can be generated for each image forthe step of image analysis:

TABLE 1 Element Season 1 Season 2 Season 3 Season 4 Element 1(C₁) P₁₁P₁₂ P₁₃ P₁₄ W_(i) = C₁*P₁₁ W_(i) = C₁*P₁₂ W_(i) = C₁*P₁₃ W_(i) = C₁*P₁₄Element 2(C₂) P₂₁ P₂₂ P₂₃ P₂₄ W_(i) = C₂*P₂₁ W_(i) = C₂*P₂₂ W_(i) =C₂*P₂₃ W_(i) = C₂*P₂₄ Element 3(C₃) P₃₁ P₃₂ P₃₃ P₃₄ W_(i) = C₃*P₃₁ W_(i)= C₃*P₃₂ W_(i) = C₃*P₃₃ W_(i) = C₃*P₃₄ Element 4(C₄) P₄₁ P₄₂ P₄₃ P₄₄W_(i) = C₄*P₄₁ W_(i) = C₄*P₄₂ W_(i) = C₄*P₄₃ W_(i) = C₄*P₄₄ TotalWseason = Wseason = Wseason = Wseason = Σ C_(i)*P_(i) Σ C_(i)*P_(i) ΣC_(i)*P_(i) Σ C_(i)*P_(i)

In Table 1, the prevalence value associated with each element and seasonis illustrated. The first subscript is the element value and the secondsubscript is the season. The P value is a measure of the strength of theassociation between the element and the season and is valued betweenzero and 1. The C value for each Element is the confidence value thatthe Element is accurately identified in the digital image.

Note that this method can be used generally to create a table relatingimages to seasons, as shown below for Table 2. The row Total fromexample Table 1 comprises the four column values under seasons 1 through4 for each row Image 1 through Image n in Table 2. Finally, the lastcolumn in Table 2 identifies which of the seasons for each image, Image1 through Image n, has obtained the highest seasonal determination value(MAX(W_(ij))) and is used as the season associated with that image.

TABLE 2 Best Season Image Season 1 Season 2 Season 3 Season 4 MatchImage 1 W₁₁ W₁₂ W₁₃ W₁₄ MAX(W_(1j)) Image 2 W₂₁ W₂₂ W₂₃ W₂₄ MAX(W_(2j))Image 3 W₃₁ W₃₂ W₃₃ W₃₄ MAX(W_(3j)) Image 4 W₄₁ W₄₂ W₄₃ W₄₄ MAX(W_(4j))Best Image MAX(W_(i1)) MAX(W_(i2)) MAX(W_(i3)) MAX(W_(i4)) Match

In Table 2, the weighting for each image in an image set for eachseasonal opening in a template is shown as calculated in the equationsand Table 1 above. The largest value in a season column specifies thebest image match for that season. The largest value in an image rowspecifies the best seasonal match for an image. If it is desired tomatch a template to a selected image, a template with a seasonal openingcorresponding to the best seasonal match for the selected image can bechosen (largest value in the image row). If it is desired to match animage to a selected template, an image with the maximum value for thatseason can be chosen (largest value in the season column). Referring toFIG. 7, the association set is provided in step 712, elements in animage found in step 713, and weights assigned in step 714 to each foundelement. A combination of different weights can be used to determine theassociated season.

The method of a preferred embodiment of the present invention isemployed to create a calendar, as illustrated in FIG. 6. The step 600 ofproviding a template includes the step of providing a calendar and thecalendar includes a template for each month of the calendar year. Themonths are each associated with a season and the template for each monthincludes a digital image selected from a seasonal group associated withthe month. The calendar is a bi-fold calendar having an upper page and alower page, the upper page including one or more digital images and thelower page including a grid of dates. The steps for forming such pagesthat include images and grids, which are then ordered and attached,fastened, or fixed together, are well known in the art and are notdescribed further.

In an alternative illustration, a preferred embodiment of the presentinvention is employed to create the image products illustrated in FIGS.5A and 5B. In FIG. 5A a single-sheet template 510 includes an opening512 for a digital image. Customized text is employed and decorativeelements (not shown) are also included. FIG. 5B illustrates a similarimage product with two openings.

To make the image products illustrated in FIGS. 5A and 5B, a method of apreferred embodiment of the present invention searches through a set ofdigital images. Since only one season is needed for the image product(e.g. calculating a column of Table 2), the seasonal groups could be“anniversary” and “not anniversary”. If suitable images are found thatare assigned to the “anniversary” seasonal group, the image product isprovided, for example by employing the suitable image with the highestranking in the template opening.

In an alternative method, the template is selected after the digitalimages are analyzed (e.g. calculating a row of Table 2). For example, auser might have a favorite image and desire an image product thatappropriately employs and complements the favorite image. In this case,the digital image is analyzed and a season determined for the image. Acorresponding image product that has a known theme associated with theseason is provided and the selected digital image composited with thecorresponding image product and produced.

Referring to FIG. 8, in this case the provision of the digital image(step 805) is the same step as selecting the digital image (if only oneimage is provided). If multiple images are provided, some selectionstake place. Once selected, the digital image is analyzed (step 810), adate is optionally compared (step 815) and a season determined (step820). The provided digital image is composited 840 with the productassociated with the selected template and the product is produced (step845).

Combinations of the processes illustrated in the flow graphs of FIGS. 6and 8 are possible, using the determined seasons of the digital imagesin combination with desired image products to make a final image productor products that include one or more digital images.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications are effected within the spirit and scope ofthe invention.

PARTS LIST

-   24 system-   26 system-   28 system-   29 printer-   30 printer-   32 I/O-   34 processor-   35 I/O-   38 sensor-   39 memory-   40 storage-   42 storage-   44 storage-   44 comm-   48 memory-   50 interface-   52 memory-   54 system-   56 I/O-   58 I/O-   58 a I/O-   58 b I/O-   58 c I/O-   66 I/O-   68 I/O-   68 a I/O-   68 b I/O-   70 system-   72 user-   300 step-   301 step-   305 step-   306 step-   310 step-   315 step-   320 step-   325 step-   326 step-   330 step-   410 template-   412 image opening-   412A image opening-   412B image opening-   412C image opening-   414 text-   420 grid of dates-   422 background-   424 fold line-   510 template-   512 image opening-   512A image opening-   512B image opening-   514 text-   600 provide template step-   605 provide digital image step-   610 analyze digital image step-   615 compare date step-   620 determine season step-   625 sort digital images step-   630 order images step-   635 select digital images step-   640 composite images step-   645 produce product step-   712 step-   713 step-   714 step-   805 provide digital image step-   810 analyze digital image step-   815 compare date step-   820 determine season step-   802 step-   800 step-   840 composite images step-   845 produce product step-   800 select template step-   805 select image step-   810 template loop step-   815 image loop step-   820 analyze image step-   825 assign match step-   830 increment image step-   835 find maximum match value step-   840 assign image to season step-   845 increment opening step

1. A computer implemented method for producing an image product, comprising the steps of: providing a digital template having one or more digital openings; providing one or more digital images each having pixels, each opening associated with one of a plurality of seasons for disposing one of the digital images therein, said one of the digital images associated with one of the plurality of seasons; automatically analyzing the pixels of the one or more digital images to determine which one of the plurality of seasons is depicted by each of the one or more digital images; for each template opening, automatically disposing a digital image therein having associated therewith a same season as the season associated with that template opening; and producing the template having the digital images disposed in the one or more openings therein.
 2. The method of claim 1, further comprising the steps of storing a date and location corresponding to each of the one or more digital images.
 3. The method of claim 1, further comprising the step of providing a template having a plurality of openings all associated with a single season.
 4. The method of claim 1, wherein the step of producing the templates includes a step selected from steps consisting of printing, storing, displaying, selling, and transporting the template having the one or more digital images disposed therein.
 5. The method of claim 1, wherein the step of providing a template includes the step of providing a template that includes openings associated with seasons selected from the group consisting of winter, spring, summer, autumn (fall), dry season, harmattan season, rainy (wet) season, monsoon season, and holiday seasons.
 6. The method of claim 5, wherein the holiday seasons are selected from the group consisting of a birthday, an anniversary, Valentine's Day, National Day, Thanksgiving, Christmas, and New Year's Day.
 7. The method of claim 1, further including the steps of: storing each of the one or more digital images in a seasonal group corresponding to a determined season for each of the one or more digital images; ranking the digital images in each seasonal group in order of a determined quality of each of the digital images in the seasonal group; and preferentially selecting digital images having a higher rank.
 8. The method of claim 1, wherein the step of automatically analyzing the pixels further includes the step of providing an association set including items selected from the group consisting of objects, colors, textures, and shapes, wherein each of said objects, colors, textures, or shapes has one of the plurality of seasons associated therewith.
 9. The method of claim 8, wherein the step of automatically analyzing the pixels includes the step of finding in each of the one or more digital images an item from the association set and assigning each of the one or more digital images to a season corresponding to the item from the association set.
 10. The method of claim 9, wherein a plurality of items in the association set are found in a single digital image.
 11. The method of claim 13, wherein the step of automatically analyzing includes the step of providing a weighted value to each found item.
 12. The method of claim 11, wherein the weighted value indicates a likelihood that each found item matches the item in the association set.
 13. The method of claim 11, wherein the weighted value indicates a prevalence of each found item in its associated season.
 14. The method of claim 11, wherein an item is associated with more than one season.
 15. The method of claim 1, wherein the step of providing a digital template includes the step of providing a digital calendar template.
 16. The method of claim 15, further comprising the steps of associating a month of the calendar with one of the plurality of seasons and wherein the template having the digital images disposed therein includes a digital image associated with a season associated with the month.
 17. A computer implemented method for producing an image product, comprising the steps of: providing one or more digital images each having pixels; analyzing the pixels of the one or more digital images for determining one of a plurality of seasons depicted by each of the one or more digital images; providing a digital template having a plurality of digital openings each associated with one of the plurality of seasons, each for disposing one of the one or more digital images therein; for each digital template opening, selecting a digital image depicting a same season as that associated with said each digital template opening; and producing the image product comprising the digital template having the digital images disposed in the openings therein.
 18. The method of claim 24, further comprising the step of sorting the one or more digital images into one or more seasonal groups corresponding to the determined seasons of each of the digital images.
 19. A computer implemented method comprising: generating and storing a list of elements, each of the elements including an associated season and a weight value indicating a strength of association between each element and its associated season; providing a plurality of digital images; and analyzing each of the digital images, including detecting a presence of one or more of the elements in each of the digital images, calculating a confidence value for each element detected in said each of the digital images, the confidence value indicating a likelihood that the detected element is actually an element in the list of elements, and calculating a season determination value for each of the digital images, the season determination value based on the weight value for each of said one or more of the elements found in each of the digital images and based on the confidence value for each element detected in said each of the digital images.
 20. The computer implemented method of claim 19, further comprising the step of storing a season identifier with each of the digital images for identifying an associated season for said each of the digital images, the step of storing a season identifier based on a magnitude of the season determination value. 